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A METHOD OF SELECTING TARGETS AND GENERATING 
FEEDBACK IN OBJECT TRACKING SYSTEMS 



5 CROSS-REFERENCE TO RELATED APPLICATIONS 

This is a continuation of Application No. 09/505,449, filed February 16, 

2000. 



BACKGROUND OF THE INVENTION 
1 0 The present invention relates to an improved video capture and display 

system. 

Object tracking systems are used for auto focusing, auto zoom, and other 
functions of video cameras. Likewise, object tracking systems are used for auto zoom, 
brightness adjustment, contrast adjustment, and highlighting functions of video display 

15 systems. Imaging devices in video cameras capture images by converting light into 
electrical signals through a photoelectric process. On the other hand, video display 
devices utilize electrical signals to control a light source producing an image. Object 
tracking systems use a number of known methods to process the electrical signals 
corresponding to an image and to extract a group of signals representing a designated 

2 0 target of the object tracking system. The group of signals representing the target object is 
relocated and "followed 11 in the succession of images or frames making up a video 
sequence. 

To initiate the tracking process, a target must be initially identified for the 
object tracking system. Object tracking systems may automatically identify a target 
2 5 object based on an assumed usage model for the video capture or display device. For 

example, the object of interest will most likely to be located near the center of the image 
and may be the larger object in the vicinity. However, automatic target designation is 
complex and is problematic if the object of interest is not centrally located in the image, 
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is one of a plurality of similar objects in the vicinity, or if there is little contrast between 
the object of interest and the surroundings. 

User designation of the target object avoids some of the limitations and 
complexities of automatic target designation. Nishimura et al., U.S. Patent 
5 No. 5,63 1 ,697, for VIDEO CAMERA CAPABLE OF AUTOMATIC TARGET 

TRACKING, describe a system in which the user designates the target by aligning an 
electronic marker generated in the viewfinder with the image of the intended target object 
and then depressing a switch to activate the target tracking processor. While less 
complex and more flexible than fully automatic target designation, the system provides 

10 only for user designation of the target object's location. 

To separate the group of signals representing the target object from the 
larger group of signals representing the image, target tracking systems utilize information 
about the size the target object, as well as its location. Automatic or manual methods 
may be used to designate the target's size once its location has been manually designated. 

15 Automatic determination of the target object's size is complex and often unreliable, 

especially in crowded or low contrast images. To reduce the complexity of the tracking 
system and increase the reliability of target identification, the system may permit the user 
to designate the target's size. For example, the target object's size may be designated by 
drawing an "electronic" box bounding the target object. However, drawing a box around 

2 0 a moving target while operating a video camera is not easily accomplished and even 

locating a target in a viewfinder when there is little contrast between the target object and 
the surroundings or when the target is located at a distance may be difficult for a user. 

What is desired, therefore, is a method of designating a target object for an 
object tracking system that allows the user to easily locate the target object and reliably 

2 5 designate its location and size for the tracking system. Further, a method of notifying the 
user of the continuing performance of the tracking system is desired. 



SUMMARY OF THE INVENTION 

The present invention overcomes the aforementioned drawbacks of the 
prior art by providing an object tracking system that includes initiating the object tracking 
system, magnifying the image, selecting an object of interest in an image, and designating 
the object as the target all while the object tracking system is activated. In this manner it 
is easier for the user of the object tracking system to select the target, especially if the 
target tends to be moving fast. 

In another aspect of the object tracking system an image is magnified and 
an object is designated as the target for tracking by the tracking system. In response to 
designating the object the scale of the image is automatically changed. In one 
embodiment, this permits the user to zoom in on the image, select the target more easily, 
and then automatically zoom out, which permits easier selection of the target. 

In yet another aspect of the object tracking system monitors a level of 
confidence that the tracking system is properly tracking the target. In response to a 
change in the level of confidence the magnification of the image visible to the operator is 
altered. 

The foregoing and other objectives, features and advantages of the 
invention will be more readily understood upon consideration of the following detailed 
description of the invention, taken in conjunction with the accompanying drawings. 

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS 

FIG. 1 is a block diagram of a video camera employing the method of the 
present invention. 

FIG. 2 is a flow chart of object tracking functionality. 

FIG. 3 is an exemplary viewfinder image of the video camera of FIG. 1 . 

FIG. 4 is a magnified view of the exemplary viewfinder image of FIG. 2. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 

Referring to FIG. 1, a video camera according to the present invention 
includes video capture and display systems. The video camera includes a lens unit 10, an 
imaging device 12 which may be a charge coupled device (CCD) or metal oxide 
5 semiconductor (MOS) array, an AID (analog to digital) converter 14, a signal processor 16, 
a driver 18 to drive the imaging device 12, and a D/A (digital to analog) converter 20. The 
imaging device 12 includes a plurality of photoelectric elements arranged in a two- 
dimensional array on one surface. The light of an optical image is focused on the 
photoelectric elements of the imaging device 12 by the lens unit 10. Each photoelectric 

1 0 element converts the light from the image that is striking the element into an analog 

electrical signal. Periodically, the driver 18 connects elements of the imaging device 12 to 
the A/D converter 14 causing the electrical signals produced by the elements of the imaging 
device 12 to be converted to digital signals which are read by the signal processor 16. The 
signal processor 16 converts the digital signals to composite video signals representing the 

15 luminance and chrominance of the light striking each element of the imaging device 12. 

The composite video signals are output to the D/A converter 20 for conversion to the analog 
signals of the camera's video output 22. The video output 22 may be transmitted or 
recorded and is displayed on a viewfinder 24 so that the user can visualize the images being 
transmitted or recorded by the camera. 

2 0 The video camera also includes a target tracking processor 26 that processes 

the output of the signal processor 16 to extract signals representing the target object from the 
signals representing the image in the succession of images or frames making up the video 
sequence. The video camera includes a zoom processor 28, a cursor generator 30, and a 
controller 32. The camera may also include a zooming motor 34 and an auto focus motor 36 

2 5 driving the zooming and focusing mechanisms of the lens unit 10. As the lens 10 zooms in 
or out, the optical image is magnified on the imaging device 12 producing a magnified 
image at the video output 22 and in the display of the viewfinder 24. It is to be understood 
that magnification may include the result of zooming in or zooming out. On the other hand, 
zooming may be accomplished by processing signals from a region of the imaging 
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device 12. The zoom processor 28 may extract signals from a portion of the imaging 
device 12 and calculate zoom information from those signals to electronically magnify a 
portion of the image that is displayed in the viewfmder 24. 

Traditional video tracking systems include the user zooming in or zooming 
5 out until the object is clearly visible within the display and then the user initiates the 

tracking functionality. Thereafter, the user then selects the target to be tracked by the video 
tracking system. The present inventor came to the realization that while a workable 
technique for tracking targets, it is clumsy because by the time the user initiates the tracking 
functionality and then selects the target, the target may have already moved out of the field 

10 of view. This is especially true when the object takes up a significant portion of the display, 
such as when under extreme zoom. To overcome these limitations the present inventors 
have developed a new interface for a tracking system. Referring to FIGS. 2 and 3, first the 
tracking functionality is initialized by any appropriate technique. Next, the target is located, 
sized, and positioned on an electronic viewfinder 24 for accurately selecting and subsequent 

15 tracking. A cursor 42 is generated by the cursor generator 30 and is superimposed on the 
image in the electronic viewfinder 24. The cursor 40 may be in a fixed location in the 
viewfinder 24 (e.g., centered) and positioned over the image of the target object by 
movement of the camera which effectively moves the viewfinder image behind the 
cursor 42. On the other hand, the cursor may be movable and electronically positioned in 

2 0 the viewfinder image by operation of a joystick, touch pad or other pointing device 38 (refer 
to FIG. 1). A target designator switch 39 facilitates signaling the controller 32 that a target 
is being designated for tracking. The target designator switch 39 and the pointing device 38 
may be combined into a single control element permitting the user to conveniently position 
a movable cursor 42 and designate the target. A movable cursor 42 permits the user to 

25 designate target objects that are not centered in the image. To initiate tracking, the user 

aligns the cursor 42 to superimpose the cursor 42 on the target object and actuates the target 
designator switch 39. Preferably, the target designator switch 39 and the pointing device 38 
are combined by touch sensitive viewfinder 24. In this manner, the user may simply target 
and select the target to be tracked by a single action. The physical activity used on the video 
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camera for targeting and selecting {e.g., buttons and controls) are preferably different than 
that used for zooming, which permits the targeting and selecting functions to be placed into 
a "ready" state. In the case of the touch screen viewfinder 24, while touching the viewfinder 
24 may be used for many different functions, after initialing the tracking functionality the 
5 touch screen viewfinder' s next function is automatically set to targeting and selecting of the 
target. Accordingly, after zooming a single action of touching the viewfinder 24 screen will 
perform targeting and selecting in a fast and efficient manner before the target, especially if 
moving fast, is no longer displayed on the viewfinder 24. 

In one embodiment of the present invention, actuation of the target designator 

10 switch 39 causes the controller 36 to signal the zoom processor 28 to begin zooming. The 

zoom processor 28 performs the zoom calculation or actuates the zoom motor 34 causing the 
zoom lens unit 34 to zoom in, magnifying the image in the viewfinder 24. Referring to 
FIG. 4, the image in the viewfinder 24 is magnified until the target object 44 is easy to view 
and designate. The user then designates the target object by actuating the target 

1 5 designator switch 39 for a second time. This may likewise be performed by a touch 

sensitive viewfinder 24 where after initiating tracking functionality the user touches the 
screen to start zooming and touches the screen a second time to designate the target. The 
first zooming may be automatic, if desired. Designation of the target causes the target 
tracking processor 26 to extract the signals representing a target designated at the location of 

2 0 the cursor (or touch on the screen) from the signals of the image in a scan of the imaging 

device 12. A target may be identified in any manner. The target tracking processor 26 then 
"follows" this group signals representing the target object in subsequent scans of the 
imaging device 12 to track the target as the video sequence progresses. 

In another embodiment of the present invention, actuation of the target 

2 5 designator switch 39 causes the controller 36 to signal the target tracking processor to 

initiate tracking and the zoom processor 28 to begin zooming in. When the image of the 
tentative target is easy to view in the viewfinder 24 the user designates the target object a 
second time by actuating the target designator switch 39. The target object is tentatively 
identified for the target tracking processor 26 by the first actuation of the target designator 
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switch 39 but the information about the size and location of the target is refined and verified 
in the magnified image where the target object is more visible. 

The zoom processor 28 may then zoom out electronically or by actuating the 
zoom motor 34 to return the zoom to the original setting or some intermediate setting 
5 established by the user. The present inventors realized that zooming in on the target object 
would make it easier to locate the target object if it is at a distance from the camera or in low 
contrast or crowded surroundings. Further, automatically zooming the image provides a 
convenient method of establishing the target. This may likewise be performed with a touch 
sensitive display. 

1 0 The algorithm applied by the target tracking processor 26 may include a 

measurement of the level of confidence that the target is being correctly tracked. If the level 
of confidence should decrease, the tracking processor 26 may cause the zoom processor 28 
to begin zooming in. The changing magnification of the image in the viewfinder 24 notifies 
the user that the confidence level of the algorithm is low. Increasing the magnification of 

15 the image may cause the algorithm's confidence level to increase. On the other hand, if the 
algorithm cannot automatically reacquire the target object, increasing the magnification of 
the image will facilitate relocation of the target object by the user who may re-designate the 
target for the tracking system by actuating the target designator switch 39. 

While tracking a target periodically the target may be obscured or otherwise 

2 0 not visible on the viewfinder 24 for a period of time. This may occur, for example, when 
another object comes in between the target and the camera for a period of time. When the 
target is not detected by the target tracking processor, the system may automatically switch 
into an alternative global tracking motion mode to predict where the object should be. In 
this manner, when the target is no longer obscured or otherwise not visible on the viewfinder 

25 24 the target tracking processor will have an increased change of identifying the target to 
continue tracking. In this manner, the system is less likely to loose track of the target. 

When the confidence of the system is sufficiently low (or losing the target 
totally) the present inventors came to the realization that most likely the target is no longer 
visible within the viewfinder 24. Accordingly, in such an event the system may 
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automatically zoom out (or in) to present the viewer with an image having a greater (or 
lesser) field of view. In most instances, the target will be visible to the user within the new 
field of view image. The user then may select the target within this field of view in the 
manner described above, as desired. Alternatively, after the zooming in response to the 
5 confidence level the system may exit the tracking functionality. 

Yet in another aspect of the present invention, the present inventors came to 
the realization that control over the amount of magnification provided by the system for the 
selection of the target should operate in an intuitive and efficient manner. The intuitive 
system developed by the present inventors includes the user selecting a target object for 

10 tracking, in any manner, such as those previously described. Preferably, after selecting the 
target object the tracking system tracks the target object. In the event that the user triggers 
another target object selection within a predetermined time period, then the system will 
cease object tracking, if object tracking was already initiated, and change the magnification 
of the image on the display. The same or a different object may be selected for the another 

15 target object selection. For example, the magnification for the target objection selection 

may be performed by, for example, clicking a mouse twice (or multiple times) or touching a 
display twice (or multiple times). After changing the magnification of the image, the system 
then permits the user to select a target object for tracking. Again, if the user triggers another 
target object selection within a predetermined time period, then the system will cease object 

2 0 tracking, if object tracking was already initiated, and change the magnification of the image 
on the display (such as increase magnification). This process is repeated until the user does 
not trigger another target object selection within a predetermined time period. This permits 
magnification of the image until the target can be easily selected for tracking. The target 
selected for tracking is preferably the last selected object. After tracking is initiated, and 

2 5 preferably the predetermined time period has elapsed, the system may return to a previous 
magnification, if desired. 

While the tracking system has been described in relation to a video camera, 
the teachings are equally applicable to selection and tracking when viewing video, from any 



source, such as for example, a video cassette recorder, digital tape, analog tape, compact 
disc, and digital video disc. 

All the references cited herein are incorporated by reference. 

The terms and expressions that have been employed in the foregoing 
specification are used as terms of description and not of limitation, and there is no intention, 
in the use of such terms and expressions, of excluding equivalents of the features shown and 
described or portions thereof, it being recognized that the scope of the invention is defined 
and limited only by the claims that follow. 



